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Aircrew  training  devices  (ATDs)  are  often  equipped  with  sophisticated 
hardware  and  software  capabilities  that  permit  a  simulator  instructor  (SI) 
to  control,  monitor,  record,  and  fabricate  flight  simulation  training 
missions.  These  advanced  instructional  features  (AIFs)  reflect  the  primary 
role  of  the  ATD  as  a  flight  trainer.  The  training  value  of  an  ATD  is  a 
function  of  the  degree  to  which  it  simulates  a  particular  aircraft  and  the 
way  in  which  it  is  used  as  an  instructional  device. 


AIFs  are  costly  to  implement  and  in  order  to  justify  these  costs, 
several  questions  must  be  answered.  How  frequently  are  AIFs  used?  How 
easy  are  they  to  use?  Are  simulator  instructors  adequately  trained  to  use 
AIFs?  Do  AIFs  have  significant  training  value? 


This  report  describes  the  third  phase  of  a  three-phase  project 
designed  to  obtain  answers  to  these  questions  by  surveying  simulator 
instructors  from  the  Air  Force  Major  Commands.  An  on-site  survey  was 
administered  to  159  Sis  assigned  to  replacement  training  units  and 
continuation  training  units  at  principal  Air  Training  Command  (T-5), 
Strategic  Air  Command  (T-4,  B-52  Weapon  System  Trainer,  FB-lllA),  and 
Tactical  Air  Command  (F-4G,  A-10)  ATD  facilities.  The  survey  requested 
background  information,  along  with  five  seven-point  rating  scales  for 
evaluating  each  of  14  AIFs.  Written  comments  concerning  the  14  AIFs  or  the 
ATD  were  solicited. 


Based  on  the  utility  and  utilization  ratings,  the  T-5  and  T-4 
trainers  were  the  most  favorably  evaluated  devices  surveyed.  They  were 
followed,  in  order,  by  the  F-4G  simulator,  B-52  WST,  and  A-10  simulator. 
Mission  control  features  (e.g.,  freeze,  reset,  and  programmed  and  manual 
threat  control)  were  generally  rated  high  in  utility  and  utilization, 
whereas  briefing  features  (e.g,,  instructor  tutorial,  recorded  briefing, 
demonstration)  and  feedback  features  (e.g.,  hard  copy,  record/playback, 
electronic  warfare  performance  scoring)  tended  to  receive  lower  ratings. 


The  level  of  AIF  use  was  affected  somewhat  by  hardware  and/or 
software  unreliability,  implementation  time,  functional  limitations,  and 
design  deficiencies.  The  perceived  training  value  of  a  feature  was  the 
most  important  determiner  of  its  use. 
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PREFACE 

This  project  was  conducted  to  satisfy  requirements  of  Air  Force  Human 
Resources  Laboratory  Technical  Planning  Objective  3,  the  thrust  of  which  is 
aircrew  training  effectiveness.  The  general  objective  of  this  thrust  is  to 
identify  and  demonstrate  cost-effective  simulator  training  strategies  and 
training  equipment  capabilities  for  use  in  developing  and  maintaining  the 
combat  readiness  of  Air  Force  aircrew  members  at  optimum  cost.  More 
specifically,  the  research  was  conducted  under  the  Air  Combat  Training 
Research  subthrust,  the  goal  of  which  is  to  provide  a  technology  base  for 
training  high  level  and  quickly  perishable  skills  in  simulated  combat 
environments.  Work  Unit  1123-02-34,  Development  and  Evaluation  of  Advanced 
Instructional  Features,  addressed  a  portion  of  this  subthrust.  Or.  Wayne 
Waag  (AFHRL/OTU)  was  the  Contract  Monitor  and  Dr.  Donald  J.  Polzella  and  Dr. 
David  C.  Hubbard,  under  contract  to  the  University  of  Dayton  Research 
Institute,  were  the  Co-Investigators. 

This  effort  was  jointly  coordinated  by  the  Air  Force  Human  Resources 
Laboratory,  Operations  Training  Division,  Williams  Air  Force  Base,  Arizona; 
the  Simulator  System  Program  Office  (SimSPO)  of  the  Air  Force  Systems 
Command,  Aeronautical  Systems  Division  (AFSC/ASD),  Wright-Patterson  Air 
Force  Base,  Ohio;  Headquarters  Air  Training  Command,  Randolph  Air  Force 
Base,  Texas;  Headquarters  Tactical  Air  Command,  Langley  Air  Force  Base, 
Virginia;  and  Headquarters  Strategic  Air  Command,  Offutt  Air  Force  Base, 
Nebraska.  The  author  gratefully  acknowledges  the  assistance  of  the 
following  individuals: 
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AIRCREW  TRAINING  DEVICES:  UTILITY  AND  UTILIZATION 
OF  ADVANCED  INSTRUCTIONAL  FEATURES 
(PHASE  III  -  ELECTRONIC  WARFARE  TRAINERS) 


I.  INTRODUCTION 

An  Aircrew  Training  Device  (ATD)  serves  two  functions.  First,  it  is  a 
ground-based  substitute  aircraft  that  permits  student  flight  crews  to  fly 
in  a  safe  and  carefully  controlled  environment.  More  importantly,  an  ATD 
is,  as  its  name  implies,  a  teaching  machine  that  is  designed  to  facilitate 
the  acquisition  of  flight  crew  skills.  In  order  to  fulfill  this  second 
function,  an  ATD  is  equipped  with  sophisticated  hardware  and  software 
capabilities  that  permit  a  simulator  instructor  (SI)  to  control,  monitor, 
and  fabricate  simulator  training  missions.  These  capabilities,  which  are 
listed  in  Table  1,  are  known  as  advanced  instructional  features  (AIFs). 

The  list  was  compiled  from  several  sources,  but  it  was  drawn  primarily  from 
Semple,  Cotton,  and  Sullivan's  (1981)  extensive  report  describing  the  AIF 
capabilities  of  various  military  and  commercial  devices. 


Table  1.  Advanced  Instructional  Features 


BRIEFING  FEATURES 

Recorded  Briefing  permits  simulator  instructor  to  provide  a  student  with 
information  about  the  simulator  and/or  a  training  mission  through 

audiovisual  media  presentation.^ 

Demonstration  permits  simulator  instructor  to  demonstrate  optimal  aircrew 
performance  by  means  of  prerecording  and  subsequently  playing  back 

segments  of  simulated  flight.  ^ 


Instructor  Tutorial  provides  simulator  instructor  with  self-paced 
programmed  instruction  in  the  capabilities  and  use  of  the  simulator.^ 


Table  1.  (Continued) 


TRAINING  MANAGEMENT  FEATURES 

Total  System  Freeze  permits  simulator  instructor  to  suspend  simulated 
flight  by  freezing  all  system  parameters,^ 

Reset  permits  simulator  instructor  to  return  the  simulated  aircraft  to  a 
stored  set  of  conditions  and  parameters.^ 

Crash  and/or  Kill  Override  permits  simulator  instructor  to  allow 


simulated  flight  to  continue  without  interruption  following  a  "crash 
"kill." 

Automated  Adaptive  Training  is  the  computer-controlled  variation  in 
task  difficulty,  complexity,  and/or  sequence  based  on  student’s 
performance. 

Programmed  Mission  Scenarios  are  computer-controlled  standardized 


training  missions  based  on  preprogrammed  event  sequences. 

Manual  Mission  Control  permits  simulator  instructor  to  modify  programmed 
scenarios  during  a  training  session.^ 

VARIATION  OF  TASK  DIFFICULTY  FEATURES 

Automated  Malfunction  Insertion  permits  simulator  instructor  to 


preprogram  a  sequence  of  aircraft  component  malfunctions  and/or 
emergency  conditions.^ 


iMMtm 


Table  1.  (Continued) 


Environmental  permits  simulator  instructor  to  vary  environmental 
conditions  such  as  wind  direction  and  velocity,  turbulence,  temperature 
and  visibility. 

Dynamics  permits  simulator  instructor  to  vary  flight  dynamics 
characteristics,  such  as  stability,  system  gain,  cross-coupling,  etc. 

Motion  permits  simulator  instructor  to  provide  a  student  with  platform 
motion  system  cues  such  as  roll,  pitch,  and  yaw. 

Flight  System  Freeze  permits  simulator  instructor  to  simultaneously 
freeze  flight  control  and  propulsion  systems,  position,  altitude, 
and  heading. 

Position  Freeze  permits  simulator  instructor  to  simultaneously  freeze 
latitude  and  longitude. 

Attitude  Freeze  permits  simulator  instructor  to  simultaneously  freeze 
pitch,  bank,  and  heading. 

Parameter  Freeze  permits  simulator  instructor  to  freeze  any  one  or  a 
combination  of  flight  parameters.^ 

MONITORING  FEATURES 

Closed  Circuit  TV  permits  simulator  instructor  to  monitor  student's 
behavior  from  the  instructor  console. 

Repeaters /Annunciators  provide  simulator  instructor  with  replicas  or 
analog  representations  of  flight  instruments  and  controls  at  the 
instructor  console. 


Table  1.  (Concluded) 


Instructor  Console  Displays  permit  simulator  instructor  to  monitor 
parameters  and  procedures  at  the  instructor  console  by  means  of 

alphanumeric  and/or  graphic  CRT  displays  of  performance  data.^ 
Automated  Performance  Alert  provides  simulator  instructor  with  visual 


and/or  auditory  signals  that  indicate  specific  performance  def iciencies. 

FEEDBACK  FEATURES 

Record/Playback  permits  simulator  instructor  to  record  and  subsequently 
play  back  a  segment  of  simulated  flight.® 

Automated  Performance  Feedback  provides  a  student  with  visual  and/or 
auditory  signals  (including  verbal  messages)  that  identify  performance 
deficiencies. 

Automated  Voice  Controller  is  the  computer-based  technology  that 


simulates  the  role  of  a  controller  by  combining  speech  generation 
speech  recognition,  and  situation  awareness  capabilities. 

Hard  Copy  provides  a  record  of  alphanumeric  and/or  graphic 


performance  data.' 


Performance  Scoring  provides  a  metric  that  summarizes  aircrew  task 
performance  during  a  simulated  mission.  ® 


Theso  -r.  ’.jiieil  m  the  Phase  III  questionnaire. 


It  appears  that  military  ATDs  are  more  often  treated  as  substitute 
aircraft  than  as  teaching  machines.  A  recent  report  by  the  United  States 
General  Accounting  Office  (1983)  concluded  that  the  Armed  Services  have  not 
sufficiently  analyzed  their  training  requirements  for  simulators.  Nor  have 
they  adequately  incorporated  simulators  into  their  training  programs.  In 
justifying  the  purchase  of  ATDs,  the  Services  have  focused  instead  on 
"duplicating  the  actual  weapon  systems  and  their  surroundings. . .with  little 
reference  to  how  the  devices  could  meet  training  needs"  (p.4). 

By  providing  AIF  capability,  simulator  manufacturers  apparently 
recognize  that  the  training  value  of  an  ATD  is  determined  not  only  by  the 
degree  to  which  it  faithfully  mimics  a  particular  aircraft,  but  also  by  the 
way  that  it  is  used  (Caro,  1973).  Previous  research  suggests  that 
effective  AIF-based  simulator  training  is  practicable  (see  Polzella,  1983, 
p.8).  However,  instructional  features  are  expensive  to  implement, 
especially  those  features  that  require  the  development  of  complex  software. 
In  order  to  justify  these  costs,  some  questions  concerning  the  present  and 
potential  utility  and  utilization  of  AIFs  should  be  answered:  How 
frequently  and  easily  are  AIFs  used?  Are  simulator  instructors  adequately 
trjined  to  use  AIFs?  Do  AIFs  have  significant  training  value? 

The  present  investigation  was  conducted  at  the  request  of  the 
Simulator  System  Program  Office  (SimSPO)  of  the  Air  Force  Systems  Command, 
Aeronautical  Systems  Division  (AFSC/ASD)  in  order  to  answer  these 
questions.  The  specific  objectives  of  this  investigation  were: 

1.  To  document  and  compare  the  utilization  (i.e.,  frequency  and  ease 
of  use)  of  AIFs. 

2.  To  document  and  compare  the  utility  (i.e.,  training  value)  of 

AIFs. 

3.  To  compare  the  utility  and  utilization  patterns  of  AIFs  in 
replacement  (e.g.,  basic,  primary,  lead-in,  initial,  transition)  and 
continuation  (e.g.,  advanced,  follow-on,  refresher,  operational)  training 
units. 

A  broader  objective  of  this  investigation  was  to  provide  a  database  that 
could  be  helpful  both  in  defining  the  requirements  for  ATD  procurements  and 
in  developing  future  ATD  training  programs. 

These  objectives  were  to  be  accomplished  in  three  phases  by  means  of  a 
survey  of  simulator  instructors  from  the  Air  Force  Major  Commands 
(MAJCOMs).  Phases  I  and  II  have  already  been  completed,  and  the  results  of 
those  surveys  are  documented  in  two  earlier  reports  (Polzella,  1983,  1985). 

The  subjects  in  Phase  I  were  134  simulator-qualified  Instructor  Pilots 
and  Weapons  Director  Instructors  (WDIs)  assigned  to  Replacement  Training 
Units  (RTUs)  and  Continuation  Training  Units  (CTUs)  at  F-4E,  F-4G,  F-15,  A- 
10,  and  E-3A  Tactical  Air  Command  (TAC)  training  sites.  The  results 
indicated  that  most  TAC  Sis  received  little  training  in  Air  use  and  that 
most  features  were  not  used  very  often.  Several  factors  appeared  to  have 
contributed  to  the  low  usages:  (a)  hardware  and/or  software  unreliability, 
(b)  time-consuming  implementation,  (c)  functional  limitations,  and  (d) 
design  deficiencies.  The  results  of  a  multiple  regression  analysis 


Indicated  that  ease  of  use  and  training  value  accounted  for  most  of  the 
variability  in  the  frequency-of-use  ratings. 


The  utility  and  utilization  of  particular  AIFs  differed  both  as  a 
function  of  ATD  and  of  training  unit.  For  example,  features  such  as  freeze 
and  reset  were  generally  used  more  often  during  RTU  missions,  whereas 
programmed  mission  scenarios  were  generally  used  more  often  during  CTU 
missions.  These  differences  appeared  to  reflect  differences  in  the 
respective  training  missions.  Thus,  RTU  missions  characteristically 
include  a  series  of  discrete  procedural  exercises,  whereas  lengthier 
scenarios  are  common  during  CTU  missions. 

The  subjects  in  Phase  II  were  273  simulator-qualified  instructor 
pilots  (IPs),  instructor  flight  engineers  (IFEs),  and  instructor  radar 
navigators  (IRNs)  assigned  to  Air  Training  Command  (ATC;  T-37,  T-38), 
Military  Airlift  Command  (MAC;  C-5A,  C-141,  C-130,  CH-3,  HH-53)  or 
Strategic  Air  Command  (SAC;  FB-lllA)  ATD  training  sites. 

The  most  striking  difference  between  the  Phase  I  and  Phase  II  results 
was  in  the  overall  magnitude  of  the  ratings.  In  comparison  to  the  TAC  Sis, 
the  ATC,  MAC,  and  SAC  Sis  used  AIFs  more  often,  found  them  easier  to  use, 
received  more  training  in  their  use,  and  considered  AIFs  to  be  more 
important  for  training.  The  results  suggested  that  TAC's  training  program 
for  Sis  is  less  extensive  and  less  structured  than  those  of  the  other 
MAJCOMs. 

The  level  of  AIF  use  among  ATC,  MAC,  and  SAC  Sis  was  affected  somewhat 
by  hardware  and/or  software  unreliability,  ir^ilementation  time,  functional 
limitations,  and  design  deficiencies.  However,  training  value  appeared  to 
be  the  most  important  determiner  of  AIF  use. 

Based  on  the  results  of  Phases  I  and  II,  it  was  recommended  that 
future  procurement  of  AIFs  be  preceded  by  a  detailed  front  end  analysis 
that  clearly  relates  AIF  capability  to  training  needs.  The  analysis  should 
consider  all  known  training  applications  of  the  simulator  as  well  as  any 
major  constraints  in  the  operational  environment.  During  procurement,  AIF 
specifications  should  be  prepared  to  meet  user  needs  and  to  ensure 
equipment  reliability.  After  operational  deployment,  the  user  should 
provide  adequate  instructor /operator  training  in  AIF  use. 

Phase  III,  which  is  described  in  this  report,  extended  the  survey  to 
electronic  warfare  instructors  from  ATC,  TAC,  and  SAC. 


II.  METHOD 


Subjects 

The  subjects  in  Phase  III  were  155  simulator-qualified  electronic 
warfare  instructors  (lEW),  weapon  systems  officers  (WSOs),  aerial  gunnery 
instructors  (lAGs),  and  radar  navigator  instructors  (IRNs).  The 
distribution  of  Sis  among  the  various  ATD  sites  surveyed  is  shown  in  Table 
2.  Also  included  in  that  table  are  the  Sis'  mean  (and  standard  deviation) 
number  of  hours  of  instructor  experience. 


ATD 

ATD-SITE 

Type  of 
training 

T-5 

Mather 

AFB 

Basic 

T-4  (B-52) 

Castle 

AFB 

Transition 

Mather 

AFB 

Operational 

WST  (B-52) 

Castle 

AFB 

Transition 

Wurtsmith 

AFB 

Operational 

FB-lllA® 

Plattsburgh 

AFB 

Transition 

Plattsburgh  Operational 
AFB 

Pease 

AFB 

Operational 

F-4G 

George 

AFB 

Replacement 

George 

AFB 

Continuation 

A-10 

Davis- 
Monthan  AFB 

Replacement 

England 

Continuation 

Questionnaire 

The  questionnaire  that  was  used  to  survey  the  instructors  is  shown  in 
Appendix  A.  Although  it  is  similar  to  those  used  during  Phases  I  and  II 
(see  Polzella,  1983,  Appendix  A;  1985,  Appendix),  several  important 
modifications  were  incorporated. 

The  first  page  of  the  questionnaire  requested  information  concerning 
flying  and  simulator  experience,  the  type  of  training  in  ATD  operations 
received  by  the  Sis,  a  description  of  a  typical  simulator  training  session, 
and  general  comments  and/or  recommendations. 

The  second  page  of  the  questionnaire  included  a  list  of  14  AIFs  (drawn 
from  the  list  in  Table  1)  and  their  definitions,  and  a  space  next  to  each 
feature  that  was  used  to  indicate  the  operational  status  of  that  feature 
(e.g.,  no  such  capability,  and  capability  present  but  unreliable,  and 
capability  present  and  reliable). 

On  subsequent  pages  were  five  questions  concerning  the  utility  and 
utilization  of  each  feature. 

1.  How  often  have  you  used  it? 

2.  How  difficult/easy  is  it  to  use? 

3.  How  inadequate/adequate  was  the  training  you  received  in  its  use? 

4.  As  presently  implemented,  how  useful  is  it? 

5.  How  potentially  useful  is  it? 


For  the  fifth  question.  Sis  were  to  assume  that  they  had  no  prior 
knowledge  of  the  features  and  to  base  their  responses  on  the  feature 
definitions  alone.  This  question  was  included  in  order  to  achieve  a  common 
basis  for  comparison  among  all  Sis,  This  was  not  otherwise  possible 
because  the  various  ATDs  were  not  similarly  equipped. 

Responses  to  each  question  were  indicated  by  checking  the  appropriate 
interval  along  a  seven-point,  successive-category  rating  scale.  (On 
certain  questions  a  0-interval  was  included  for  indicating  "not 
applicable.")  The  intervals  of  each  scale  were  labeled  with  descriptive 
adjectives  in  order  to  facilitate  responding  and  to  help  interpret  the 
ratings.  Additional  space  was  provided  for  comments. 

Procedure 

The  questionnaire  was  administered  on-site  to  various  sized  (N  =  5  to 
10)  groups  of  Sis.  The  Sis  were  briefed  on  the  purpose  of  the 
investigation  and  copies  of  the  questionnaire  were  distributed  and 
thoroughly  reviewed  prior  to  being  filled  out.  For  the  most  part,  the 
questionnaire  was  self-explanatory.  However,  Question  3  (How 
inadequate/adequate  was  the  training  you  received?)  required  some 
additional  instruction.  For  this  question,  the  Sis  were  asked  to  rate  each 
feature  twice.  The  first  rating  assessed  the  training  received  in  the 


operation  of  that  feature,  whereas  the  second  rating  assessed  the  training 
received  in  the  effective  use  of  that  feature  (in  terms  of  student 
outcome). 

The  questionnaire  could  be  completed  in  approximately  30  minutes. 


III.  RESULTS 


Table  3  lists  the  14  AIFs  that  were  included  in  the  questionnaire 
along  with  their  definitions.  The  table  also  lists  a  mnemonic  code  for 
each  feature,  which  will  be  used  in  subsequent  tables.  The  AIF 
capabilities  of  the  various  ATDs  are  shown  in  Table  4. 

The  Phase  III  questionnaire  yielded  the  following  data  from  each  SI: 

1.  Number  of  hours  of  flying  and  simulator  experience. 

2.  Type  of  training  received. 

3.  Description  of  a  typical  training  session. 

4.  Assessment  of  the  operational  status  of  each  AIF. 

5.  Ratings  of  the  AIFs  on  each  of  the  five  questions  (including  the 
two  ratings  of  Question  3). 

6.  Comments. 

The  data  were  classified  by  ATD  (T-5,  F-46,  A-10,  T-4  WST,  FB-lllA),  level 
of  training  (e.g.,  transition,  operational).  Question  (1  through  5),  and 
AIF  (1  through  14).  The  resulting  data  matrix  was  unbalanced  due  to 
differences  in  the  numbers  of  Sis  and  in  the  AIF  capabilities  of  the 
various  ATDs  (see  Tables  2  and  4).  In  most  cases,  this  necessitated 
analyzing  the  data  from  each  ATD  separately. 

Descriptive  statistics  were  computed  for  type  of  training  received  by 
the  Sis  and  for  their  assessment  of  the  operational  status  of  each  AIF. 
Multivariate  analyses  of  variance  were  used  to  analyze  the  ratings  of  each 
feature  across  the  first  four  questions.  The  data  from  Question  5  were 
analyzed  separately.  The  multivariate  model  was  based  on  a  two-factor 
mixed  design  in  which  level  of  training  was  the  between-subjects  factor, 

AIF  was  the  within-subjects  factor,  and  the  ratings  on  the  first  four 
questions  were  the  dependent  variables.  Missing  data  were  deleted  "list- 
wise,"  i.e. ,  subjects  who  did  not  rate  a  feature  on  all  four  questions  were 
eliminated  from  the  analyses. 

Approximate  F-values,  derived  from  Wilks'  lambda,  were  used  to  test 
the  overall  multivariate  significance  of  each  effect  in  the  model,  i.e., 
AIF,  level-of-training,  and  the  AIF  by  level-of-training  interaction,  while 
univariate  Fs  were  used  to  test  the  significance  of  these  effects  for  each 
of  the  four~questions  separately.  Tukey  honestly  significant  difference 
(HSD)  values  were  computed  for  each  univariate  analysis.  These  values  were 
used  to  determine  significant  differences  between  particular  ratings,  e.g.. 


U-a. 


Table  3.  Advanced  Instructional  Features  Included  in  the 
Phase  III  Questionnaire 


Code  Feature 


IT  Instructor  Tutorial  -  provides  the  instructor  with  self-paced 

prograttmed  instruction  in  the  capabilities  and  use  of  the  simulator. 

R  Reset  -  permits  instructor  to  "return"  the  siitwlated  aircraft  to  a 
stored  set  of  conditions  and  parameters. 

TSF  Total  System  Freeze  -  permits  instructor  to  interrupt  and  suspend 
simulated  flight  by  freezing  all  system  parameters. 

PF  Partial  Freeze  -  permits  instructor  to  freeze  various  flight 

parameter  combinations  such  as  altitude,  heading,  position,  attitude, 
flight  system,  etc. 

RB  Recorded  Briefing  -  permits  instructor  to  provide  student  with 

information  about  a  structured  training  session  through  audio/visual 
media  presentation. 

0  Demonstration  -  permits  instructor  to  demonstrate  optimal  electronic 
warfare  procedures  by  prerecording  and  subsequently  playing  back  a 
simulated  engagement. 

RP  Record/Playback  -  permits  instructor  to  record  and  subsequently 
playback  a  segment  of  simulated  flight. 

AMI  Automated  Malfunction  Insertion  -  permits  instructor  to  preprogram  a 
sequence  of  aircraft  component  malfunctions  and/or  emergency 
conditions. 

HC  Hard  Copy  -  provides  a  record  of  alphanumeric  and/or  graphic 
performance  data  for  debriefing  purposes. 

PTC  Programmed  Threat  Control  -  conpu ter- control  led  standardized 
training  sessions  based  on  preprogrammed  event  sequences. 

MTC  Manual  Threat  Control  -  permits  instructor  to  modify  threat  scenarios 
during  a  training  session. 

PRM  Procedures  Monitoring  -  permits  instructor  to  monitor  discrete 

actions  performed  by  the  student  in  accordance  with  a  procedurally 
defined  checklist. 

PAM  Parameters  Monitoring  -  permits  instructor  to  monitor  various 
instrument  readings,  control  settings,  aircraft  states,  or 
navigational  profiles. 


EWS  Electronic  Warfare  Performance  Scoring  -  provides  a  performance 
metric  that  summarizes  the  outcomes  of  EW  engagements. 


training  value  of  manual  threat  control  vs.  training  value  of  programmed 
threat  control,  frequency  of  RTU  use  of  reset  vs.  frequency  of  CTU  use  of 
reset. 


The  interrelations  among  the  utility  and  utilization  ratings  were 
determined  by  means  of  correlation  and  regression  analyses.  First, 
intercorrelations  were  computed  between  the  ratings  of  each  feature  across 
the  five  questions.  Second,  multiple  linear  regression  analyses  were  used 
to  determine  those  variables  which  significantly  predicted  the  frequency  of 
AIF  use.  Three  potential  predictors  were  evaluated:  the  ease  of  AIF  use, 
the  adequacy  of  training  received  (a  composite  variable  representing  the 
average  of  the  two  ratings  on  Question  3),  and  AIF  usefulness  (a  composite 
variable  representing  the  average  of  the  ratings  on  Questions  4  and  5). 
Missing  data  were  deleted  "list-wise"  from  these  analyses. 


Air  Training  Command 


T-5  Trainer 

Training  mission.  The  T-5  ATD  is  a  sophisticated  generic  trainer  for 
primary  level  electronic  warfare  skills.  The  typical  training  session 
lasts  3  to  4  hours  and  includes  a  15-minute  prebriefing  of  mission 
objectives,  a  2  1/2-to  3  1/2-hour  mission,  and  a  30-minute 
debriefing/critique.  A  complete  mission,  from  takeoff  to  landing,  normally 
requires  the  student  to  search  for,  identify,  and  determine  the  parameters 
of  electronic  warfare  (EW)  signals  and  select  appropriate  countermeasures. 
The  instructor's  role  is  to  (a)  monitor  student  progress  for  speed  and 
accuracy,  (b)  freeze  and  offer  feedback  verbally  and  through  demonstration, 
and  (c)  reset  as  required. 

Training  of  Sis.  On  the  average,  formal  classroom  instruction 
accounted  for  28X  of  the  initial  training  for  the  T-5  Sis  (SO  =  29%).  Only 
11%  of  the  Sis  reported  having  received  any  refresher  training,  and  nearly 
all  of  that  training  was  characterized  as  informal. 

AIFs.  The  operational  status  of  each  AIF  on  the  T-5  trainer  is 
summarized  in  Table  5.  A  small  percentage  of  Sis  indicated  that  there  are 
operational  problems  with  recorded  briefing  and  programmed  threat  control. 
Otherwise,  all  available  features  appear  to  operate  reliably.  The  table 
shows  that  a  substantial  proportion  of  Sis  apparently  have  never  operated 
auto  malfunction  insertion  (0.32),  manual  threat  control  (0.58),  or 
electronic  warfare  performance  scoring  (0.28).  (Note:  These  proportions 
include  those  Sis  who  indicated  "no  such  capability"  for  these  features.) 

Utilization  and  utility  ratings.  The  ratings  for  the  T-5  Sis  are 
summarized  in  Table  6.  Means  and  standard  deviations  are  listed  for 
available  features  under  Questions  1  to  4  and  for  all  features  under 
Question  5  (potential  usefulness).  The  nwltivariate  analysis  of  variance 
revealed  a  significant  overall  effect  of  AIF,  F(45,463.85)  =  4.25,  p  < 

.001.  Each  univariate  F  (df  =  9,107)  was  also  significant,  £  <  .001. 


Table  6.  T-5  Trainer:  Mean  Ratings  (and  Standard  Deviations) 

of  the  Frequency  of  Use,  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and  Potential 
Training  Value 


Frequency  of  use  was  highest  for  programmed  threat  control,  procedures 
monitoring,  and  parameters  monitoring.  These  features  were  used  at  an 
average  rate  of  at  least  five  times  each  mission.  Frequency  of  use  was 
lowest  for  auto  malfunction  insertion,  hard  copy,  and  manual  threat 
control.  These  features  were  used  significantly  less  often  than  most  of 
the  AIFs  (Tukey  HSD  qj=  2.08).  With  the  exception  of  manual  threat 

control,  ease  of  use  was  uniformly  high  for  all  AIFs  (Tukey  HSD  = 

1.13).  The  training  received  by  the  Sis  was  apparently  adequate  despite 
the  lack  of  both  formal  classroom  instruction  and  refresher  training.  (See 
previous  section.)  This  was  not  the  case  for  manual  threat  control, 
however.  Training  in  its  operation  and  in  its  effective  use  was  judged 
to  be  adequate.  Manual  threat  control  received  significantly  lower  ratings 
on  these  variables  than  did  all  other  AIFs  (p  <  .05).  A  similar  pattern 
emerged  for  the  training  value  ratings.  Most  of  the  features  were  judged 
to  be  at  least  very  useful,  whereas  manual  threat  control  was  rated  the 
least  useful  of  all  AIFs  (Tukey  HSD  qj  =  1.51). 

The  separate  analysis  of  the  potential  training  value  ratings 
(Question  5)  also  yielded  a  significant  effect  of  AIF,  F(13,231)  =  8.65,  £ 

<  .001.  Every  feature  was  judged  to  have  at  least  moderate  potential 
usefulness  (including  manual  threat  control);  however,  those  features  that 
are  presently  unavailable  (i.e.,  instructor  tutorial,  partial  freeze,  etc.) 
were  generally  rated  lower  than  were  those  AIFs  that  are  already 
implemented  (Tukey  HSD  Qg  =  1.42). 

Interrelations  among  utilization  and  utility  ratings.  Table  7  shows 
the  intercorrelations  among  the  ratings  of  each  feature  on  each  of  the  five 
questions.  All  the  coefficients  were  positive  and  significant,  £  <  .01. 

Thus,  a  feature's  rating  on  any  question  can  be  predicted  with  greater  than 
chance  accuracy  given  its  rating  on  any  other  question.  For  example,  the 
more  useful  a  feature  was,  the  more  frequently  it  was  used,  the  easier  it 
was  to  use,  the  more  adequate  was  the  training  in  its  use,  and  the  greater 
was  its  potential  training  value.  However,  these  predictions  would  not  be 
equally  precise.  The  coefficients  of  determination  (i.e.,  the  squared  r 
values)  ranged  from  .07  (FREQUSE/TRECD(2) )  to  .64  (TRECD(1 ) /TRECD(2) )  over 
the  entire  matrix. 

Table  8  summarizes  the  results  of  a  multiple  linear  regression  analysis 
in  which  the  frequency  of  AIF  use  was  predicted  from  a  linear  combination  of 
ease  of  use,  adequacy  of  training  received  (a  composite  variable  representing 
the  average  of  the  two  ratings  on  Question  3),  and  training  value  (a 
composite  variable  representing  the  average  usefulness  and  potential 
usefulness).  The  table  indicates  that,  together,  the  predictor  variables 
accounted  for  approximately  30%  of  the  variability  in  the  frequency-of-use 
ratings.  However,  the  only  significant  predictor  was  training  value. 

Comments.  Most  of  the  comments  concerning  the  T-5  ATD  were  favorable. 
However,  several  instructors  noted  that  instructor  training  and  programming 
support  are  inadequate;  consequently,  many  of  the  advanced  capabilities  of 
the  T-5,  such  as  computer-aided  instruction,  are  not  fully  utilized.  A 
significant  operational  deficiency  is  that  the  T-5  is  too  slow  to  adequately 
monitor  rapidly  performed  procedural  checklists. 


Note  All  correlations  are  significant,  jd  <  .01, 


Table  8.  T-5  Trainer:  Multiple  Linear  Regression  of 

Frequency  of  Use  on  Ease  of  Use,  Adequacy  of 
Training  Received,  and  Training  Value 


strategic  Air  Command 


T-4  Trainer 

Trai^ning  mission.  The  T-4  ATD  provides  both  transition  and 
operafionaT'frafhTng  for  B-52  electronic  warfare  officers.  The  typical 
training  session  lasts  2  to  3  hours  and  includes  a  15-minute  prebriefing  of 
mission  objectives  and  interference  checks,  a  2-hour  mission,  and  a  15- 
minute  debriefing/critique.  The  transition  and  operational  missions  are 
highly  similar.  Both  missions  require  the  student  to  recognize  and 
counteract  a  series  of  threats  encountered  during  high-level,  low-level, 
and  over-water  penetrations  into  enemy  territory.  Various  malfunctions  and 
emergencies  are  distributed  throughout  the  mission.  Although  a  mission  can 
proceed  under  computer  control,  the  T-4  SI  can  modify  mission  flow  and 
content  through  the  discretionary  use  of  manual  insertions,  freezes,  and 
resets. 

Traini^ng^  of_SIs.  The  T-4  Sis  received  both  formal  and  info-^mal 
instructTdn.'  'FdrmaT  classroom  instruction  accounted  for  38%  of  r-4  RTU 
Sis'  initial  training  (SO  =  31%),  and  70%  of  T-4  CTU  Sis'  initial  training 
(SO  =  25%).  Also,  45%  of  the  RTU  Sis  and  38%  of  the  CTU  Sis  reported 
having  received  refresher  training  (approximately  once  within  the  preceding 
year).  However,  26%  of  the  RTU  refresher  training  and  80%  of  the  CTU 
refresher  training  were  characterized  as  formal  classroom  instruction. 

AlFs.  The  operational  status  of  each  AIF  on  the  T-4  trainer  is 
summanzed  in  Table  9.  All  available  features,  except  recorded  briefing, 
appear  to  operate  reliably. 

Utilization  and  utilUy^  ratings.  The  ratings  by  the  T-4  RTU  and  CTU 
Sis  are'suriimar fzed  in  fables  lif  and  11,  respectively.  Means  and  standard 
deviations  are  listed  for  available  features  under  Questions  1  to  4  and  for 
all  features  under  Question  5  (potential  usefulness).  The  multivariate 
analysis  of  variance  revealed  a  significant  overall  effect  of  AIF, 
F(25,410.13)  =  4.34,  £  <  .001.  Except  for  training  value,  each  univariate 
f  (df  =  5,114)  was  also  significant,  £  <  .001.  Neither  level  of  training, 
nor  the  AIF  by  level  of  training  interaction  was  significant  at  the 
multivariate  level. 

Frequency  of  AIF  use  was  high  overall.  The  average  feature  was  used 
at  least  two  to  four  times  a  mission.  Procedures  and  parameters  monitoring 
were  used  most  often,  threat  control  (programmed  and  manual)  and  freeze, 
slightly  less  so.  Recorded  briefing  was  used  hardly  at  all  (Tukey  USD  = 

.88).  Ease  of  use  was  also  high  overall,  although  the  means  were  not 
statistically  equivalent  (Tukey  HSD  qj  =  .84).  The  training  received  by 

the  Sis  (in  both  the  operation  and  effective  use  of  the  features)  was 
judged  to  be  "very  adequate."  Moreover,  each  feature  (except  recorded 

briefing,  as  rated  by  the  transition  Sis)  was  judged  to  be  "very  useful." 

The  separate  analysis  of  the  potential  training  value  ratings  revealed 
a  significant  effect  of  AIF,  F(13,337)  =  16.29,  £  <  .001.  However,  neither 
the  effect  of  level  of  training  nor  the  AIF  by  level  of  training 


Table  10. 


T-4  Transition  Training:  Mean  Ratings  (and  Standard 
Deviations)  of  the  Frequency  of  Use,  Ease  of  Use, 
Adequacy  of  Training  Received,  Training  Value,  and 
Potential  Training  Value 
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T-4  Operational  Training:  Mean  Ratings  (and  Standard 
Deviations)  of  the  Frequency  of  Use,  Ease  of  Use, 
Adequacy  of  Training  Received,  Training  Value, 
and  Potential  Training  Value 


interaction  was  significant,  F(l,26)  =  3.13,  £  >  .05  and  F(13,337)  =  1.15, 
p  >  .05,  respectively.  Potential  usefulness  was  fairly  high  overall. 
Prograimed  and  manual  threat  control  received  the  highest  ratings,  whereas 
instructor  tutorial,  partial  freeze,  recorded  briefing,  hard  copy,  and 
electronic  warfare  performance  scoring  received  only  moderate  ratings 
(Tukey  HSD  qj  =  1.52). 

Interrelations  among  utilization  and  utility  ratings.  Table  12  shows 
the  intercorrelations  among  the  ratings  of  each  feature  on  each  of  the  five 
questions.  Most  of  the  coefficients  were  positive  and  significant.  The 
matrix  suggests,  for  example,  that  the  more  useful  a  feature  was,  the  more 
frequently  it  was  used,  the  easier  it  was  to  use,  the  more  adequate  was  the 
training  in  its  use,  and  the  greater  was  its  potential  training  value.  It 
should  be  noted,  however,  that  not  all  variables  were  significantly 
correlated.  For  example,  although  ease  of  use  and  adequacy  of  training 
received  were  positively  correlated,  neither  variable  was  related  to 
frequency  of  use.  The  matrix  reflects  considerable  variability  in  the 
level  of  predictability  among  the  variables.  The  coefficients  of 
determination  ranged  from  .00  (FREQUSE/EASEUSE)  to  .53  (TVALUE/PTVALUE) . 

Table  12.  T-4  Trainer:  Matrix  of  Intercorrelations  Among 

Frequency  of  Use,  Ease  of  Use,  Adequacy  of  Training 
Received,  Training  Value,  and  Potential  Training  Value 
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DEPENDENT  VARIABLE:  Frequency  of  AIF  Use 


MULTIPLE  R:  .53  STD.  ERROR  OF  EST.:  1.28 

MULTIPLE  R-SQUARE:  .28 

ANALYSIS  OF  VARIANCE: 


Sum  of  Squares 

df 

Mean  Square  F-Ratio 

£ 

REGRESSION 

117.9240 

3 

39.3080  23.954 

.0000 

RESIDUAL 

297.0165 

181 

1.6410 

Standard 

Predictor 

variable 

Coefficient 

Standard 

error 

regression 
coefficient  t 

£ 

EASEUSE 

.0400 

.0934 

.0294  .428 

.6690 

TRECD 

-.2972 

.1225 

-.1757  -2.426 

.0163 

TVALUE 

.7720 

.0917 

.5679  8.419 

.0000 

(CONSTANT) 

2.5259 

.7709 

Training  value  was  clearly  the  most  important  predictor.  However,  adequacy 
of  training  also  contributed  significantly  to  the  equation  (even  though  it 
was  negatively  related  to  the  dependent  variable).  Ease  of  use  did  not 
contribute  significantly. 

Comments.  The  T-4  ATD  was  one  of  the  most  highly  regarded  devices 
surveyed.  In  fact,  many  lEWs  preferred  it  to  the  more  sophisticated  B-52 
Weapon  System  Trainer  (B-52  WST;  see  below).  One  frequent  criticism  was 
that  modifications  in  T-4  hardware  and  software  have  not  kept  pace  with 
those  of  the  aircraft. 

B-52  Weapon  System  Trainer  -  Defensive  Stations 

Training  mission.  The  B-52  Weapon  System  Trainer  (WST)  is  one  of  the 
most  sophisticated  aTDs  in  the  Air  Force  inventory  (Stein,  1984).  It  can 
provide  training  for  the  entire  six-man  B-52  crew  via  three  separate 
instructor  consoles:  the  flight  instructors  console  (pilot,  co-pilot),  the 
navigation  instructors  console  (navigator,  radar  navigator),  and  the 
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defensive  avionics  systems  instructors  console  (electronic  warfare  officer, 
gunner) . 

The  WST  can  operate  in  either  independent  or  integrated  mode,  depending  on 
whether  one  or  all  of  the  crew  stations  are  active  at  a  given  time.  The 
integrated  capability  makes  the  WST  particularly  useful  for  training  crew 
coordination.  The  typical  independent  WST  training  session  (defensive)  is 
similar  to  that  of  the  T-4  except  that  the  WST  has  more  sophisticated  threat 
library,  weapons,  real-time  randomness,  maneuvering,  monitoring  and  mission 
generation  capabilities.  The  integrated  session,  which  includes  briefing, 
mission,  and  debriefing,  may  last  up  to  five  hours.  The  independent  mission 
normally  lasts  up  to  two  hours. 

The  emergency  war  order  (EWO)  mission  provides  a  context  for  both 
transition  and  operational  (continuation)  training.  The  integrated  EWO 
mission  includes  equipment  checks  and  malfunction  evaluations,  takeoff,  air 
refueling,  high-level  flight,  descent  into  enemy  territory,  defense  of 
aircraft  in  various  encounters  (airborne  interceptors,  ships,  land-based 
threats),  monitoring  of  EW  equipment,  ECM,  malfunction  analyses,  climb  and 
withdrawal,  and  landing.  During  independent  missions,  the  instructors  use 
multiple  malfunctions,  freezes,  and  resets  in  order  to  work  on  specific 
problem  areas  (e.g.,  malfunction  analyses,  signal  recognition,  jamming). 

Training  of  Sis.  On  the  average,  formal  classroom  instruction 
accounted  for  only  8%  of  the  initial  training  for  the  RTU  Sis  (SO  =  11%) 
and  25%  for  the  initial  training  of  the  CTU  Sis  (SO  =  29%).  Only  one  WST 
SI  reported  having  received  any  refresher  training. 

AIFs.  The  operational  status  of  each  AIF  on  the  B-52  WST  is 
summarized  in  Table  14.  There  appear  to  be  operational  problems  associated 
with  all  available  features  except  auto  malfunction  insertion.  The  least 
reliable  AIFs  (i.e.,  those  AIFs  that  were  called  "unreliable"  by  at  least 
0.30  of  the  Sis)  were  record/playback  (0.73),  programmed  threat  control 
(0.33),  procedures  monitoring  (0.53),  and  electronic  warfare  performance 
scoring  (0.53). 

Utilization  and  utility  relations.  The  B-52  WST  RTU  and  CTU  Sis’ 
ratings  are  summarized  in  Tables  15  and  16,  respectively.  Means  and 
standard  deviations  are  listed  for  available  features  under  Questions  1  to 
4  and  for  all  features  under  Question  5  (potential  usefulness).  The 
multivariate  analysis  of  variance  revealed  a  significant  overall  effect  of 
AIF,  F(45,432.53)  =  4.85,  £  <  .001.  Univariate  Fs  (df  =  9,100)  were 
signiTicant  for  frequency  of  use  (£  <  .001),  ease  of  use  (£  <  .001),  and 
training  value  (p  <  .001),  but  not  for  adequacy  of  training  received  (£  < 

.05).  Although  the  multivariate  effect  of  level  of  training  was  not 
significant,  F(5,3)  =  1.05,  p  <  .05,  there  was  a  weak  but  significant  AIF 
by  level  of  training  interaction,  F(45,432.53)=  1.47,  (£  <  .05). 


Frequency  of  AIF  use  was  lower  overall  than  was  the  case  for  the  T-4. 
The  average  feature  was  used  approximately  once  each  mission.  Threat 
control  (programmed  and  manual)  and  parameters  monitoring  were  used  most 
often  (at  least  two  to  four  times  a  mission),  whereas  record/playback 
(called  "unreliable"  by  73%  of  the  Sis),  hard  copy  (called  "unreliable"  by 
20%  of  the  Sis),  and  partial  freeze  were  used  significantly  less  often 
(Tukey  HSD  gj^  =  2.48).  Ease  of  use  was  fairly  high  overall.  Programmed 
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Table  14. 


B-52  WST-Defensive  Station:  The  Number  of 
Sis  (Transition  and  Operational)  Indicating 
the  Operational  Status  of  Each  AIF 


Table  15 


threat  control,  auto  malfunction  Insertion,  and  freeze  were  rated  "very 
easy,"  whereas  record /playback  was  rated  "fairly  difficult"  (Tukey  HSD  * 

1.72).  The  training  received  by  the  Sis  was  apparently  adequate  despite 
the  lack  of  both  formal  classroom  Instruction  and  refresher  training.  (See 
previous  section.)  Threat  control  (programmed  and  manual),  auto 
malfunction  Insertion,  and  total  system  freeze  were  judged  to  have  the  most 
training  value,  whereas  record/playback  and  hard  copy  were  rated  as  only 
"fairly  useful"  (Tukey  HSD  qj  «  2.00). 

The  separate  analysis  of  the  potential  training  value  ratings  revealed 
a  significant  effect  of  AIF,  F(13,169)  »  6.72,  p  <  .001.  However,  neither 
the  effect  of  level  of  training  nor  the  AIF  by  level  of  training 
Interaction  was  significant,  F  <  1.00  and  F(13,169)  »  1.25,  £  <  .05, 
respectively.  Potential  usefulness  was  faTrly  high  overall.  In  fact.  It 
was  somewhat  higher  than  the  usefulness  of  the  AIFs  as  they  are  currently 
Implemented  (Question  4).  Extremely  high  ratings  were  assigned  to  threat 
control  (programmed  and  manual),  auto  malfunction  Insertion,  and  total 
system  freeze.  Even  the  lowest  rated  features  (e.g. ,  Instructor  tutorial, 
recorded  briefing)  were  judged  to  be  at  least  moderately  useful.  (Tukey 
HSD  Qj  =  1.86). 

Interrelations  among  utilization  and  utility  ratings.  Table  17  shows 
the  Intercorre rations  among  the  ratings  of  each  feature  on  each  of  the  five 
questions.  Most  of  the  coefficients  were  positive  and  significant;  the 
only  exceptions  were  the  correlations  of  potential  training  value  with  ease 
of  use  and  adequacy  of  training  received.  The  matrix  reflects  a  low, 
although  variable,  level  of  predictability  among  the  variables.  The 
coefficients  of  determination  ranged  from  .00  (PTVALUE/TRECD(2) )  to  .23 
(TREC0(1)/TRECD(2)). 

Table  17.  B-52  WST;  Matrix  of  Intercorrelations  Among 

Frequency  of  Use,  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and 
Potential  Training  Value 


Feature 

EASEUSE 

TRECO(l) 

TRECD(2) 

TVALUE 

PTVALUE 

FREQUSE 

1.00 

EASEUSE 

.22* 

1.00 

TRECD(l) 

.30* 

.50* 

1.00 

TRECD(2) 

.30* 

.45* 

.53* 

1.00 

TVALUE 

.49* 

.41* 

.32* 

.52* 

1.00 

PTVALUE 

.35* 

.16 

.13 

.06 

.45* 

1.00 

*  £  <  .01. 
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Table  18  summarizes  the  results  of  a  multiple  linear  regression 
analysis  in  which  the  frequency  of  AIF  use  was  predicted  from  a  linear 
combination  of  ease  of  use,  adequacy  of  training  received,  and  training 
value.  The  table  indicates  that,  together,  the  predictor  variables 
accounted  for  approximately  29%  of  the  variability  in  the  frequency-of-use 
ratings.  Training  value  was  clearly  the  most  important  predictor. 

However,  adequacy  of  training  received  also  contributed  significantly  to 
the  equation.  Ease  of  use  did  not. 


Table  18.  B-52  WST:  Multiple  Linear  Regression  of 

Frequency  of  Use  on  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and 
Potential  Training  Value 


DFPE’fTO’E’NT’*tf^SR I ^BLE :  Fr equenc^y  of  'AIF  'Use 


MULTIPLE  R:  .53 


STO.  ERROR  OF  EST. :  1.82 


MULTIPLE  R-SQUARE:  .29 


ANALYSIS  OF  VARIANCE: 


REGRESSION 


Sum  of  Squares  ^  Mean  Square  F- Ratio 


189.0932 


63.0311 


18.932 


.0000 


RESIDUAL 


472.7766 


3.3294 


Predictor  Standard 

variable  Coefficient  error 


Standard 

regression 

coefficient 


EASEUSE 


-.0724 


-.056 


.1222 


TVALUE 


.1175 


(CONSTANT) 


-.4946 


.6967 


Comments.  It  was  noted  above  that  many  WST  AIFs  were  described  as  "un- 
rel iable. "  There  were  several  other  criticisms  of  the  device:  certain  threats 
were  said  to  be  "out  of  date,"  thereby  limiting  effective  ECM  training. 
Moreover,  the  WST  generates  threats  probabilistically.  Some  repeatability  will 
be  included  with  the  threat  update  effort  now  in  progress.  Thus,  it  is  not 
possible  to  replicate  a  particular  threat  scenario  exactly.  Another  apparent 
problem  is  that  feedback  data,  which  are  presented  to  the  Sis  on 


multiple  CRT  "pages,"  are  not  arranged  for  rapid  access  during  dynamic 
situations. 

FB-lllA  Operational  Flight  Trainer 

The  FB-lllA  radar  navigator  instructors'  data  were  collected  during 
Phase  II.  Those  results  are  summarized  in  Appendix  B  (Tables  B-1  through 
B-5).  The  reader  is  cautioned  that,  due  to  differences  between  the  Phase 
II  and  Phase  III  questionnaires,  the  FB-lllA  results  and  those  of  Phase  III 
are  not  directly  comparable.  See  Polzella  (1985)  for  additional 
information  regarding  the  utility  and  utilization  of  the  FB-lllA 
Operational  Flight  Trainer  AIFs. 

Tactical  Air  Command 

F-4G  Simulator 

Training  mission.  The  F-4G  Advanced  Wild  Weasel  is  the  ultimate  USAF 
version  of  the  F-4  Phantom  II  fighter  aircraft.  In  addition  to  its 
offensive  capabilities,  the  F-4G  carries  ECM  sensors,  jamming  pods,  chaff 
dispensers,  anti-radiation  missiles,  and  advanced  avionics.  The  typical  F- 
4G  simulator  mission  required  students  to  demonstrate  the  various  system 
capabilities  within  a  highly  dynamic  scenario.  The  RTU  and  CTU  missions 
are  similar;  however,  CTU  missions  are  characterized  by  more  elaborate 
threat  scenarios,  whereas  RTU  missions  tend  to  stress  basic  tactical 
skills. 

A  typical  training  session  consists  of  a  15-minute  briefing  of  the 
planned  mission  elements,  a  1  1/2-hour  mission,  and  15  to  30  minutes  for 
debriefing  and  critique.  The  major  mission  segments  are  preflight, 
takeoff,  ingress  into  EW/target  area,  egress,  and  return  to  base.  Mission 
elements  normally  include  new  APR-38  functions,  surface-to-air  missile 
(SAM)  encounters,  air-to-air  intercepts,  ordnance  delivery,  and  a  full 
spectrum  of  EW  activity.  In  addition,  there  are  frequent  malfunctions, 
emergencies,  and  threats.  F-4G  Sis  prefer  to  insert  these  manually  and  use 
freeze  and  reset  as  needed. 

Training  of  Sis.  On  the  average,  formal  classroom  instruction 
accounted  for  36<  of  the  initial  training  for  the  RTU  SIS  (SO  =  34%)  and 
15%  of  the  initial  training  for  the  CTU  Sis  (SO  =  22%).  Only  two  F-4G  Sis 
(1  RTU,  1  CTU)  reported  having  received  any  refresher  training. 

AIFs.  The  operational  status  of  each  AIF  on  the  F-4G  simulator  is 
summarized  in  Table  19.  Except  for  hard  copy,  there  appear  to  be  few 
operational  problems  associated  with  the  F-4G  features.  However, 
substantial  proportions  of  Sis  apparent'-y  have  never  operated  partial 
freeze  (0.47),  auto  malfunction  insertion  (0.84),  and  hard  copy  (0.53). 
(Note:  The  proportions  include  those  Sis  who  indicated  "no  such 
capability"  for  these  features.) 

Utilization  and  utilUy  ratings.  The  ratings  for  the  F-46  RTU  and  CTU 
Sis  are  summarized  in  Tables  20  ana  21,  respectively.  Means  and  standard 
deviations  are  listed  for  available  features  under  Questions  1  to  4  and  for 


Table  21.  F-4G  Simulator  Operational  Training:  Mean 
Ratings  (and  Standard  Deviations)  of  the 
Frequency  of  Use,  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and  Potential 
Training  Value 


EASEUSE  TRECDd 


TRECDf^  TVALUE  PTV 


Feature  FREQUSE 
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all  features  under  Question  5  (potential  usefulness).  The  multivariate 
analysis  of  variance  revealed  a  significant  overall  effect  of  AIF,  F(25,350.7) 
8.32,  £  <  .001.  Each  univariate  £  (df  =  5,98)  was  also  significant,  £< 

.001.  The  analysis  also  revealed  a  significant  overall  effect  of  level  of 
training,  F(5,l)  =  1458.03,  £  <  .05;  however,  none  of  the  univariate  Fs 
were  significant  in  this  case,  £  <  .05.  Finally,  there  was  a  significant 
AIF  by  level  of  training  interaction,  F(25,350.7)  =  1.97,  £  <  .01. 

Univariate  interactions  were  significant  for  frequency  of  use  (£  <  .05), 
and  training  value  (£  <  .05). 

A  moderate  level  of  frequency  of  AIF  use  was  observed  overall.  The 
average  feature  was  used  only  once  each  mission.  Manual  threat  control  was 
used  with  considerable  regularity  (five  to  seven  times  each  mission), 
whereas  hard  copy  (called  “unreliable"  by  28%  of  the  Sis)  and  auto 
malfunction  insertion  were  rarely  or  never  used  (Tukey  HSD  =  1.30).  The 

AIF  by  level  of  training  interaction  apparently  reflected  the  fact  that  RTU 
Sis  used  manual  threat  control  more  often  than  they  used  all  other 
features,  whereas  CTU  (i.e.,  operational)  Sis  used  manual  threat  control, 
programmed  threat  control,  reset,  and  freeze  (total  and  partial)  at 
statistically  equivalent  rates  (Tukey  HSD  =  1.93).  While  there  appeared 

to  be  no  particular  difficulties  in  using  any  of  the  features,  some 
features  (e.g.,  reset,  total  system  freeze)  were  rated  easier  to  use  than 
others  (e.g.,  hard  copy,  manual  threat  control),  Tukey  HSD  =  1.12.  The 

RTU  and  CTU  Sis  rated  the  training  they  received  (both  in  the  operation  and 
effective  use  of  the  features)  as  adequate  (Tukey  HSD  =  1.19,1.29).  It 

is  interesting  to  note  that  the  ratings  of  the  RTU  Sis  were  somewhat  lower 
than  those  of  the  CTU  Sis,  despite  the  fact  that  the  RTU  Sis  apparently 
received  more  formal  training.  (See  "Training  of  Sis.")  The  training 
value  ratings  were  fairly  high  overall.  The  average  feature  was  judged 
"very  useful."  The  highest  ratings  were  assigned  to  manual  threat  control, 
reset,  and  total  system  freeze.  Auto  malfunction  insertion  and  hard  copy 
received  significantly  lower  ratings  than  did  all  other  features  (Tukey  HSD 
=  1.07).  The  significant  AIF  by  level  of  training  interaction  was 

apparently  due  to  a  greater  degree  of  variability  among  the  CTU  Sis' 
ratings  than  among  those  of  the  RTU  Sis  (Tukey  HSD  g^  =  1.59). 
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The  separate  analysis  of  the  potential  training  value  ratings  revealed 
a  significant  effect  of  AIF,  F(13,387)  =  13.85,  £  <  .001.  However,  neither 
the  effect  of  level  of  training  nor  the  AIF  by  level  of  training 
interaction  was  significant,  F  <  1.00  and  F  <  1.00,  respectively. 

Potential  usefulness  was  fairly  high  overall.  Every  feature  was  judged  to 
be  at  least  fairly  useful.  Manual  threat  control,  reset,  and  total  system 
freeze  received  the  highest  ratings,  whereas  instructor  tutorial,  recorded 
briefing,  demonstration,  and  auto  malfunction  insertion  received 
significantly  lower  ratings  (Tukey  HSD  g^^  =  1.26). 

Interrelations  among  utilization  and  utility  ratings.  Table  22  shows 
the  intercorrelations  among  the  ratings  of  each  feature  on  each  of  the  five 
questions.  Most  of  the  coefficients  were  positive  and  significant. 
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FREQUSE 

1. 00 

EASEUSE 

.04 

1. 00 

TRECD(l) 

.15 

.45** 

1. 00 

TRECD(2) 

.28** 

.39** 

.64** 

I.OO 

TVALUE 

.71** 

.19* 

.30** 

.37** 

1.00 

PTVALUE 

.52** 

• 

O 

00 

.22** 

.30** 

.70** 

*£  <  .05. 

**£  <  .01. 


Interestingly,  however,  neither  the  adequacy  of  training  received  (in 
operating  the  features)  nor  the  ease  of  using  the  features  was  related  to 
frequency  of  use.  Overall,  the  matrix  reflects  a  variable  level  of 
predictability  among  the  variables.  The  coefficient  of  determination 
ranged  from  .00  (FREQUSE/EASEUSE)  to  .50  (FREQUSE/TVALUE) . 

Table  23  summarizes  the  results  of  a  multiple  linear  regression 
analysis  in  which  the  frequency  of  AIF  use  was  predicted  from  a  linear 
combination  of  ease  of  use,  adequacy  of  training  received,  and  training 
value.  The  table  indicates  that,  together,  the  predictor  variables 
accounted  for  approximately  47%  of  the  variability  in  the  frequency-of-use 
ratings.  However,  the  only  significant  predictor  was  training  value. 

Comments.  There  were  numerous  criticisms  of  the  F-4G  simulator,  most 
of  which  related  to  fidelity  and  operational  reliability.  Apparently,  most 
instructors  are  self-taught,  and  some  are  not  familiar  with  the  full  range 
of  capabilities.  Many  Sis  characterized  the  simulator  as  a  "procedural 
trainer"  rather  than  a  "full-mission  simulator,"  which  it  is  designed  to 
be.  Various  recommendations  were  made:  record/playback  capability,  rear 
cockpit  visual  display,  electronic  warfare  and  weapons  scoring,  imaging  of 
electronic  missile  launches,  and  better  and  more  structured  instructor 
training. 

A-IO  Simulator 

Training  mission.  The  A-IO  is  a  heavily  armed  (and  armored)  close  air 
support  attack  aircraft.  As  presently  configured,  the  A-IO  pilot  also 
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Table  23.  F-4G  Simulator:  Multiple  Linear  Regression  of 
Frequency  of  Use  on  Ease  of  Use,  Adequacy  of 
Training  Received,  and  Training  Value 


DEPENDENT  VARIABLE:  Frequency  of  AIF-Use 
MULTIPLE  R:  .68  STD.  ERROR  OF  EST. :  1.16 

MULTIPLE  R-SQUARE:  .47 


ANALYSIS  OF  VARIANCE: 


Sum  of  Squares 

Mean  Square 

F-Ratio 

£ 

REGRESSION 

173.0386 

3 

57.6795 

43.078 

.0000 

RESIDUAL 

198.1654 

148 

1.3390 

Predictor 

variable 

Coefficient 

Standard 

error 

Standard 

regression 

coefficient 

t 

£ 

EASEUSE 

-.0918 

.0788 

-.0788 

-1.165 

.2457 

TRECD 

.0313 

.0824 

.0272 

.379 

.7050 

TVALUE 

.9677 

.0915 

.6815 

10.575 

.0000 

(CONSTANT) 

-.8718 

.5928 

functions  as  weapon  systems  officer.  In  this  latter  capacity,  he  is 
responsible  for  navigation,  ECM  operations,  and  target  or  threat 
acquisition  and  designation. 

Because  of  its  extensive  capabilities,  the  A-IO  simulator  has  been 
used  as  both  part-task  trainer  and  full-mission  simulator.  A  fully 
integrated  mission  includes  preflight,  takeoff,  TACAN  point-to-point 
through  a  programmed  series  of  threats,  switchology,  operation  of  radar 
warning  receiver  (RWR),  threat  recognition,  evasion,  ECM  pod  operations, 
chaff/flare  usage,  weapons  delivery,  egress,  and  return-to-base.  The  major 
difference  between  the  RTU  and  CTU  missions  is  that  CTU  Sis  devote  more 
time  to  the  integrated  mission  than  do  RTU  Sis,  whereas  RTU  Sis  devote 
relatively  more  time  to  the  training  of  particular  EW  techniques. 

Training  of  Sis.  Dn  the  average,  formal  classroom  instruction 
accounted  for  2ti  oT^ the  A-10  RTU  Sis'  initial  training  (SD  =  30%)  and  11% 
of  the  CTU  Sis'  initial  training  (SD  =  16%).  Fifty  percent  of  the  RTU  Sis 
and  35%  of  the  CTU  Sis  reported  having  received  refresher  training  (within 
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the  past  18  weeks  and  within  the  past  5  weeks,  respectively).  More  than 
one-third  of  that  training  was  characterized  as  formal  classroom 
instruction. 

AIFs.  The  operational  status  of  each  AIF  on  the  A-10  simulator  is 
summarized  in  Table  24.  Apparently  there  are  operational  reliability 
problems  associated  with  all  A-10  AIFs  except  reset  and  total  system 
freeze.  In  particular,  almost  one  out  of  four  Sis  indicated  that 
electronic  warfare  performance  scoring  was  unreliable.  The  table  also 
suggests  that  a  substantial  proportion  of  Sis  have  never  operated 
demonstration  (0.97),  record/playback  (0.39),  auto  malfunction  insertion 
(0.52),  manual  threat  control  (0.30),  (advanced)  procedures  monitoring 
(0.48),  or  (advanced)  parameters  monitoring  (0.52).  (Note:  The  "advanced" 
monitoring  capabilities  inform  the  console  operator  when  aircraft 
parameters  are  "out  of  bounds."  Also  note  that  hard  copy  is  available  on 
the  A-10  simulator;  however,  it  is  not  available  for  copying  EW  data.) 

Utilization  and  utility  ratings.  The  ratings  for  the  A-10  RTU  and 
CTU  Sis  are  summarized  in  Tables  25  and  26,  respectively.  Means  and 
standard  deviations  are  listed  for  available  features  under  Questions  1  to 
4  and  for  all  features  under  Question  5  (potential  usefulness).  The 
multivariate  analysis  of  variance  revealed  a  significant  overall  effect  of 
AIF,  F(30),446)  =  7.40,  p  <  .001.  Each  univariate  F  (df  =  6,115)  was  also 
signiTicant,  £  <  .001.  The  multivariate  effect  of  Tevel  of  training  was 
not  significant,  F(5,l)  =  54.69,  £  >  .05;  however,  there  was  a  significant 
AIF  by  level  of  training  interaction,  F(30,446)  =  2.31,  £  <  .001. 
Significant  univariate  interactions  were  found  for  frequency  of  use  (£  < 
.001),  adequacy  of  training  received  (£  <  .05),  and  training  value 
(£  <  .05) . 

A  relatively  low  level  of  frequency  of  AIF  use  was  observed  overall. 
In  fact,  it  was  lower  for  the  A-10  ATD  than  for  any  other  device  surveyed. 
The  average  feature  was  used  only  once  every  two  to  four  missions. 
Demonstration,  record/playback,  and  auto  malfunction  insertion  were  rarely 
or  never  used;  however,  total  system  freeze  was  used  with  some  regularity 
by  both  RTU  and  CTU  Sis  (Tukey  HSD  =  1.52).  Moreover,  RTU  Sis  used 

procedures  monitoring,  parameters  monitoring,  and  electronic  warfare 
performance  scoring  relatively  often  despite  problems  in  reliability.  (See 
previous  section.)  CTU  Sis  used  these  features  significantly  less  often 
(Tukey  HSD  =  2.10).  Ease  of  use  was  moderately  high  for  most  features. 

However,  significant  difficulties  in  using  demonstration,  record/plavback, 
auto  malfunction  insertion,  and  manual  threat  control  were  reported  by  both 
RTU  and  CTU  Sis  (Tukey  HSD  =  1.24).  Adequacy  of  training  received  (both 

in  the  operation  and  effective  use  of  the  features)  was  rated  as  "slightly 
adequate"  overall.  Training  was  most  adequate  for  reset  and  total  system 
freeze  and  least  adequate  for  demonstration,  record/playback,  auto 
malfunction  insertion,  and  manual  threat  control  (Tukey  HSD  g^  =  1.31, 

1.35).  The  adequacy  of  training  received  in  the  operation  and  use  of 
record/playback  was  significantly  lower  for  RTU  Sis  than  for  CIs  (Tukey 
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Table  25.  A-10  Simulator  Replacement  Training:  Mean 

Ratings  (and  Standard  Deviations)  of  the 
Frequency  of  Use,  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and 
Potential  Training  Value 


Feature  FREQUSE  EASEUSE  TRECDd 


TVALUE 


pTvAlue 


Table  26.  A-10  Simulator  Operational  Training:  Mean  Ratings 

(and  Standard  Deviations)  of  the  Frequency  of  Use, 
Ease  of  Use,  Adequacy  of  Training  Received,  Training 
Value,  and  Potential  Training  Value 


TRECD 


TVALUE 


3.0 

(1.7) 


4.5 

(1.6) 


4.3 

(1.6) 


3.6 

(2.0) 


3.1 

(2.3) 


2.6 

(1.0) 


2.9 

(1.5) 


5.0 

(1.4) 


1.3 

(0.5) 


4.0 

(1.6) 


5.3 

(1.2) 


5.0 

(0.7) 


2.2 

(2.7) 


6.0 

(0.0) 


4.5 

(3.5) 


5.7 

(1.2) 


3.3 

(2.2) 


4.3 

(1.6) 


4.7 

(1.6) 


4.4 

(1.8) 


(1.9) 


(1.6) 


(1.8) 


(1.8) 


3.2 

(1.4) 


3.7 

(1.8) 


2.7 

(2.0) 


2.5 

(2.4) 


3.8 

(1.8) 


(1.9) 


IT 

3.8 

(1.7) 

R 

5.1 

5.9 

6.2 

6.3 

5.8 

6.2 

(0.6) 

(0.7) 

(1.4) 

(0.8) 

(1.3) 

(0.9) 

TSF 

5.3 

6.1 

6.8 

6.5 

6.6 

6.5 

(0.8) 

(0.3) 

(0.4) 

(0.8) 

(0.6) 

(0.9) 

PF 

2.7 

5.1 

5.7 

4.6 

4.0 

4.6 

(1.6) 

(1.4) 

(1.1) 

(1.6) 

(1.5) 

(1.4) 

RB 

2.9 

(1.7) 

0 

1.1 

3.0 

3.3 

3.7 

2.2 

3.6 

(0.3) 

(1.6) 

(1.7) 

(1.9) 

(1.3) 

(1.6) 

RP 

2.2 

4.6 

5.2 

4.9 

3.7 

4.4 

(1.0) 

(1.1) 

(1.5) 

(1.6) 

(1.4) 

(1.5) 

AMI 

1.5 

4.1 

4.1 

3.8 

3.2 

4.0 

(1.2) 

(1.5) 

(1.8) 

(2.3) 

(0.9) 

(1.2) 

4.3 

(1.6) 


4.8 

(1.4) 


5.8 

(1.3) 


4.1 

(1.7) 


4.1 

(2.0) 


4.6 

(2.0) 


■  4.5 

(1.8) 


HSD  QgS  1.81,1.87).  The  AIFs,  as  presently  implemented,  were  rated 

"moderately  useful"  overall.  In  general,  the  training  value  ratings  were 
lower  for  the  A-10  ATD  than  for  any  other  device  surveyed.  The  most  useful 
features  were  reset  and  total  system  freeze,  whereas  demonstration, 
record/playback,  and  auto  malfunction  insertion  were  judged  to  be 
significantly  less  useful  (Tukey  HSD  .01  =  ^  .41).  RTU  Sis  rated  parameters 

monitoring  significantly  higher  in  usefulness  than  did  CTU  Sis,  whereas  CTU 
Sis  rated  record /pi ay back  significantly  higher  in  usefulness  than  did  RTU 
Sis. 


The  separate  analysis  of  the  potential  training  value  ratings  revealed 
a  significant  main  effect  of  AIF,  F(13,400)  »  13.11,  p  <  .001  and  a 
significant  AIF  by  level  of  training  interaction,  F(13,400)  =  2.42,  p  < 

.01.  The  main  effect  of  level  of  training  was  not  significant,  F(l,Il)  = 
1.57,  £  >  .05.  Potential  usefulness  was  moderately  high  overall?  Reset, 
total  system  freeze,  manual  threat  control,  and  electronic  warfare 
performance  scoring  received  the  highest  ratings,  whereas  instructor 
tutorial,  recorded  briefing,  demonstration,  record /playback,  auto 
malfunction  insertion,  and  hard  copy  received  significantly  lower  ratings 
(Tukey  HSD  qj  »  1.30).  The  AIF  by  level  of  training  interaction  was 

apparently  due  to  a  greater  degree  of  variability  among  the  CTU  Sis'  mean 
ratings  than  among  those  of  the  RTU  Sis  (Tukey  HSD  Qg  =  1.62). 

Interrelations  among  utilization  and  utility  ratings.  Table  27  shows 
the  intercorrelations  among  the  ratings  of  each  feature  on  each  of  the  five 
questions.  Except  for  the  correlation  between  potential  training  value  and 
ease  of  use,  all  of  the  coefficients  were  positive  and  significant.  In 
general,  the  level  of  predictability  was  moderate.  The  coefficient  of 
determination  ranged  from  .02  (EASEUSE/PTVALUE)  to  .45  (EASEUSE/TRECDd) ). 


Table  27.  A-10  Simulator;  Matrix  of  Intercorrelations 

Among  Frequency  of  Use,  Ease  of  Use,  Adequacy 
of  Training  Received,  Training  Value,  and 
Potential  Training  Value 


F'eature'  FREQUSE  ""ETSEUSE"  "TRECDd)  ~^gC0(2)  TVALUE  PTVMe 


FREQUSE 

1.00 

EASEUSE 

.25** 

1.00 

TRECOd) 

.25** 

.67** 

1.00 

TRECD(2) 

.31** 

.47** 

.51** 

1.00 

TVALUE 

.40** 

.55** 

.44** 

.62** 

.51**  1.00 


PTVALUE  .48** 


.13 


.18** 


.40** 


Table  28  sunmBrIzes  the  results  of  a  multiple  linear  regression 
analysis  in  which  the  frequency  of  AIF  use,  adequacy  of  training  received, 
and  training  value.  The  table  indicates  that,  together,  the  predictor 
variables  accounted  for  only  25X  of  the  variability  in  the  frequency-of-use 
ratings.  The  only  significant  predictor  was  training  value. 


Table  28.  A-10  Simulator:  Multiple  Linear  Regression  of 
Frequency  of  Use  on  Ease  of  Use,  Adequacy  of 
Training  Received,  Training  Value,  and  Potential 
Training  Value 


DEPENDENT  VARIABLE:  Frequency  of  AIF  Use 
MULTIPLE  R:  .50  STD.  ERROR  OF  EST.:  1.76 

MULTIPLE  R-SQUARE:  .25 


ANALYSIS  OF  VARIANCE: 

Sum  of  Squares  ^  Mean  Square  F- Ratio  £ 


REGRESSION 

223.3051 

3 

74.4350 

24.067 

.0000 

RESIDUAL 

671.1474 

217 

3.0928 

Predictor 

variable 

Coefficient 

Standard 

error 

Standard 

regression 

coefficient 

t 

£ 

EASEUSE 

.0255 

.1104 

.0181 

.231 

.8173 

TRECD 

.0930 

.1370 

.0585 

.678 

.4983 

TVALUE 

.6892 

.1077 

.4558 

6.399 

.0000 

(CONSTANT) 

-.0152 

.5672 

Comment.  The  A-10  simulator  was  criticized  more  often  than  was  any 
other  Atd  surveyed.  The  most  frequent  criticisms  concerned  the  device's 
lack  of  fidelity  to  the  actual  aircraft.  Several  instructors  even  believed 
that  transfer  of  training,  from  simulator  to  aircraft,  was  negative.  Many 
Sis  expressed  particular  frustration  at  the  limited  (RTU)  or  absent  (CTU) 
visual  system.  Such  deficiencies  are  ironic  when  considering  that  the  A-10 
pilot  frequently  needs  visual  contact  with  targets  and  threats  in  order  to 
fulfill  his  mission. 
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Several  other  problems  were  noted:  a  lack  of  instructor  confidence  in 
the  operational  reliability  of  the  device,  insufficient  instructor 
training,  and  difficulty  in  manually  controlling  threats.  A  few  Sis  stated 
that  the  device  was  useful  for  procedural  training,  but  many  more  of  the 
comments  were  critical. 


IV.  DISCUSSION 


For  purposes  of  discussion  the  14  AIFs  surveyed  in  Phase  III  can  be 
arranged  in  three  categories: 

Briefing  AIFs  are  designed  for  briefing  the  student  and  SI  prior  to  or 
during  a  training  mission.  Their  purpose  is  to  establish  a  learning  set 
and  increase  learning  readiness.  These  features  include 

1.  Instructor  tutorial 

2.  Recorded  briefing 

3.  Demonstration. 

Mission  Control  AIFs  include  various  features  designed  to  control  the 
structure  and  sequencing  of  tasks  within  a  training  mission.  These 
features  include 

1.  Total  system  freeze 

2.  Partial  freeze 

3.  Reset 

4.  Automated  malfunction  insertion 

5.  Programmed  threat  control 

6.  Manual  threat  control. 

Monitor  and  Feedback  AIFs  permit  the  simulator  instructor  to  monitor 
student  performance  and  provide  the  student  with  performance  feedback. 

These  features  include 

1.  Procedures  monitoring 

2.  Parameters  monitoring 

3.  Electronic  warfare  performance  scoring 

4.  Hard  copy 

5.  Record/playback. 

Briefing  AIFs 

Operational  status.  Most  of  the  devices  surveyed  had  no  briefing 
capabTTity”  The  only  exceptions  were  the  T-5  (recorded  briefing),  T-4 
(recorded  briefing,  audio  only),  and  the  A~10  (demonstration).  A  few  T-5 
lEWs  characterized  recorded  briefing  as  "unreliable,"  and  an  equal  number 
apparently  had  never  operated  the  feature.  However,  most  T-5  lEWs 
considered  recorded  briefing  to  operate  reliably.  It  was  difficult  to 
assess  the  operational  reliability  of  recorded  briefing  on  the  T-4  trainer 
and  demonstration  on  the  A-10  simulator,  since  most  of  the  Sis  had  never 
operated  these  features. 


utility  and  utilization.  The  briefing  features,  where  available, 
tended  to  receive  lower  utility  and  utilization  ratings  than  did  most  of 
the  other  features.  The  potential  training  value  ratings  suggested  that 
these  features  might  have  some  potential  usefulness  for  EW  training,  but  as 
a  group,  the  briefing  AIFs  generally  received  lower  potential  training 
value  ratings  than  did  most  of  the  other  AIFs.  Instead  of  structured 
briefings,  most  Sis  appeared  to  prefer  informal  briefings,  which  could  be 
adapted  to  the  particular  needs  of  individual  students  and  instructors. 

Mission  Control  AIFs 


Operational  status.  Those  features  that  were  manually  operated  and 
simple  to  use  (e.g.,  freeze,  reset)  appeared  to  cause  few  operational 
problems.  This  was  not  the  case  for  the  other  mission  control  AIFs. 

Manual  malfunction  insertion,  although  not  included  in  the  survey,  was 
apparently  preferred  by  most  Sis  over  auto  malfunction  insertion.  Whereas 
auto  malfunction  insertion  operated  reliably  on  the  B-52  WST,  it  was  either 
never  used  or  operated  unreliably  on  all  other  devices  surveyed.  In 
addition,  approximately  one-third  of  the  B-52  WST  Sis  characterized 
programmed  and  manual  threat  control  as  "unreliable."  Although  these 
features  appeared  to  operate  more  reliably  on  the  other  devices,  the  only 
device  for  which  no  reliability  problems  were  noted  was  the  T-4  trainer. 

Utility  and  utilization.  The  ratings  of  total  system  freeze  and  reset 
were  consistently  high.  TTTe  freezing  of  an  ATD  (in  order  to  offer 
feedback)  and  the  subsequent  resumption  of  a  training  mission  appeared  to 
be  a  critical  capability  of  all  ATDs  surveyed.  Partial  freeze,  which  was 
available  only  on  the  B-52  WST  and  TAC  ATDs,  was  used  less  frequently  but 
was  nevertheless  considered  to  be  at  least  moderately  useful  for  EW 
training. 

Auto  malfunction  insertion  was  used  with  moderate  regularity 
(averaging  about  once  a  mission)  during  ATC  and  SAC  missions.  In  contrast, 
it  was  considered  less  important  for  TAC  missions  and,  consequently,  was 
used  less  often.  Indeed,  TAC  Sis  much  prefer  the  discretionary  use  of 
malfunctions.  This  may  be  a  more  appropriate  training  strategy  in  their 
case  since  the  TAC  missions  are  typically  more  dynamic  and  less  structured 
than  those  of  ATC  and  SAC.  Consistent  with  this  strategy,  the  F-4G  lEWs 
generally  assigned  higher  ratings  to  manual  threat  control  than  to 
programmed  threat  control,  whereas  the  ATC  and  SAC  Sis  generally  assigned 
similar  and  extremely  high  ratings  to  both  features. 

Monitor  and  Feedback  AIFs 


Operational  status.  The  operational  status  of  these  features  varied  greatly 
across  the  devices  surveyed.  The  A-10  simulator  is  the  only  device  that  includes 
a  record/playback  capability.  Most  of  the  A-10  RTU  Sis  never  even  operated  the 
feature.  However,  most  of  the  A-10  CTU  Sis  did  operate  the  feature,  and  none 
reported  any  problems  in  reliability.  Over  70%  of  the  B-52  WST  Sis  indicated  that 
their  record/playback  capability  was  unreliable.  It  should  be  noted,  however, 
that  record/playback  is  not  available  from  the  defensive  station.  During  an 
integrated  training  session,  only  aircraft  position  information  is  available  to 
the  defensive  station  during  a  playback.  There  is  no  playback  of  defensive  crew 
actions.  These  indications  appeared  to  reflect  deficiencies  in  software  rather 
than  hardware.  As  noted  in  a  previous  section  of  this  report,  many  B-52  WST  Sis 
commented  that  it  was  impossible  to  replicate  a  particular  threat  scenario  on  the 
WST  because  threats  are  generated  probabilistically. 
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The  operational  reliability  of  the  monitor  and  feedback  AIFs  appeared 
to  be  greatest  for  the  T-5  and  T-4  trainers.  There  were  no  "unreliable" 
indications  for  any  of  the  features.  In  contrast,  the  reliability  of  the 
monitor  and  feedback  AIFs  on  the  B-52  WST  appeared  to  be  especially 
lacking. 

Utility  and  utilization.  In  general,  record/playback  and  hard  copy, 
where  available,  received  the  lowest  utility  and  utilization  ratings.  In 
contrast,  procedures  and  parameters  monitoring  tended  to  be  among  the 
highest  rated  of  all  features  surveyed.  This  generalization  did  not  apply 
to  the  A-10  CTU  Sis,  however,  who  assigned  relatively  low  frequency-of-use 
and  training  value  ratings  to  procedures  and  parameters  monitoring.  The 
low  ratings  were  due  in  part  to  the  large  number  of  A-10  CTU  Sis  who 
indicated  "no  such  capability"  for  these  features,  although  it  is  not  clear 
why  this  was  the  case.  The  A-10  CTU  Sis  also  assigned  low  ratings  to 
electronic  warfare  performance  scoring  because  of  problems  in  reliability. 
Overall,  the  potential  usefulness  of  the  monitor  and  feedback  features  was 
relatively  high.  The  means  ranged  from  "moderately  useful"  to  "extremely 
useful." 

Differences  Among  the  ATDs 

The  survey  revealed  many  differences  among  the  ATDs  in  terms  of  AIF 
utility,  utilization,  and  operational  status.  Some  of  these  differences 
were  noted  in  the  previous  section,  but  several  general  observations  can 
also  be  made.  An  overview  of  the  operational  status  data  (Tables  5,  9,  14, 
19,  24)  suggests  that  the  T-5  trainer  was  the  most  reliable  device, 
followed,  in  order,  by  the  T-4  trainer,  F-4G  simulator,  B-52  WST,  and  A-10 
simulator.  The  devices  were  similarly  ordered  with  respect  to  AIF  utility 
and  utilization.  The  T-5  and  T-4  trainers  tended  to  receive  the  highest 
ratings,  whereas  the  B-52  WST  and  the  A-10  simulator  tended  to  receive 
lower  ratings.  Paradoxically,  the  more  sophisticated  devices  (i.e.,  those 
devices  with  the  more  extensive  instructional  support  capabilities) 
received  the  least  favorable  evaluations.  This  paradox  may  not  be 
characteristic  of  ATDs  in  general,  but  it  leads  us  to  ask  whether  effective 
EW  training  might  be  better  achieved  through  the  use  of  part-task  or 
specialized  trainers. 

Differences  Between  the  Two  Levels  of  Training 

The  pattern  of  AIF  utility  and  utilization  was,  for  the  most  part, 
similar  across  the  two  levels  of  training.  Some  differences  were  observed 
among  the  TAG  Sis,  however.  The  F-4G  RTU  Sis  used  manual  threat  control 
more  often  than  all  other  features,  and  the  A-10  RTU  Sis  used  procedures 
monitoring,  parameters  monitoring,  and  electronic  warfare  performance 
scoring  significantly  more  often  than  did  the  A-10  CTU  Sis.  The  A-10  RTU 
Sis  also  rated  parameters  monitoring  to  be  more  useful  for  training  than 
did  the  A-10  CTU  Sis.  These  differences  were  consistent  with  the 
particular  characteristics  of  the  TAC  RTU  missions,  which  are  less 
structured  and  more  closely  monitored  than  were  the  CTU  missions.  The 
failure  to  find  RTU-CTU  differences  within  SC  probably  reflected  the  fact 
that  these  missions  were  more  similar  to  one  another  than  those  of  TAC. 


Predicting  the  Frequency  of  AIF  Use 

The  multiple  linear  regression  analyses  (see  Tables  8,  13,  18,  23,  28) 
indicated  that  at  least  25%,  and  as  much  as  47%,  of  the  variability  in  the 
frequency- of-use  ratings  could  be  explained  by  the  remaining  variables. 
Training  value  was  clearly  the  most  significant  predictor  at  every  ATD  site 
surveyed.  Indeed,  it  was  the  only  significant  predictor  at  three  of  the 
five  sites.  Adequacy  of  training  received  added  a  small  but  significant 
degree  of  predictability  to  the  T-4  and  B-52  WST  regression  equations. 

What  can  be  concluded  from  these  facts?  Unfortunately,  correlational 
findings  do  not  logically  imply  causality;  they  merely  reflect  the  presence 
of  a  relationship  between  variables.  In  this  case,  however,  it  seems 
reasonable  to  assume  that  particular  AIFs  were  used  more  frequently  because 
they  were  more  useful.  Indeed,  assuming  that  the  training  value  of  an  AIF 
did  not  affect  its  use  is  clearly  implausible. 

How  can  the  fact  be  explained  that  the  remaining  variables  (i.e. ,  ease 
of  use,  adequacy  of  training  received)  did  not  account  for  much  of  the 
variability  in  frequency  of  use?  This  fact  suggests  that  the  Sis  would  not 
avoid  using  a  particular  feature,  even  if  it  were  complicated  to  use,  as 
long  as  they  believed  that  it  would  help  accomplish  mission  objectives. 

Training  Received  by  the  Simulator  Instructors 

The  ratings  suggested  that  the  Sis  considered  their  training  to  be 
more  adequate  than  inadequate.  The  T-4  Sis  gave  the  highest  ratings, 
followed,  in  order,  by  the  T-5,  F-4G,  B-52  WST,  and  A-IO  Sis.  There 
appeared  to  be  little  difference  between  the  adequacy  of  training  received 
in  the  operation  and  effective  use  of  the  features. 

There  was  considerable  variability  in  the  amount  and  type  of  training 
received  by  the  Sis.  The  amount  of  formal  initial  training  appeared  to  be 
related  to  the  magnitude  of  the  utility  and  utilization  ratings.  For 
example,  the  T-4  and  T-5  Sis  received  the  greatest  amount  of  formal  initial 
training,  and  the  T-4  and  T-5  ATDs  were  the  most  favorably  evaluated 
devices.  The  importance  of  refresher  training  was  less  clear.  Thus,  the 
T-4  and  A-IO  Sis  received  the  greatest  amount  of  refresher  training,  but 
the  ratings  of  their  respective  devices  were  quite  different. 

Comparisons  Between  Phases  I,  II,  and  III 

Due  to  the  differences  between  the  questionnaires  used  in  Phase  III 
and  in  Phases  I  and  II,  it  is  difficult  to  directly  compare  the  ratings. 

For  example,  the  frequency-of-use  ratings  obtained  from  the  Phase  III 
questionnaire  were  considerably  higher  than  those  obtained  from  the 
previous  questionnaires.  Nevertheless,  it  cannot  be  concluded  that 
electronic  warfare  Sis  use  instructional  features  more  often  than  do  other 
Sis,  because  the  appropriate  question  and  possible  answers  were  worded 
differently  in  each  case.  Thus,  in  Phase  III  the  Sis  were  asked  in 
Question  I  "During  five  typical  missions,  how  often  did  you  use  each 
instructional  feature?"  The  possible  answers  spanned  seven  categories 
ranging  from  "never"  to  "8  or  more  times  a  mission."  In  Phases  I  and  II , 
the  Sis  were  simply  asked:  "How  often  have  you  used  each  instructional 
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feature?"  In  this  case  the  possible  answers  ranging  from  "never"  to  "most 
often."  The  other  questions  were  also  worded  differently. 

It  was  possible  to  make  certain  valid  comparisons,  however.  For 
example,  within  all  phases  of  the  survey,  the  relative  ratings  of 
particular  AIFs  were  fairly  consistent  across  all  ATDs.  That  is,  those 
features  rated  highest  (lowest)  by  one  group  of  Sis  also  tended  to  be  rated 
highest  (lowest)  by  the  other  groups.  This  suggests  that  the  overall 
pattern  of  AIF  use  is  similar  across  the  MAJ(X)Ms.  Another  valid 
generalization  was  that  the  perceived  training  value  of  a  feature  appeared 
to  be  the  single  most  consistent  determiner  of  its  use. 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


At  the  conclusion  of  Phases  I  and  II,  it  was  recommended  that  certain 
AIFs  need  to  be  made  more  reliable  and  user  friendly  before  their  training 
effectiveness  can  be  ascertained.  It  was  also  recommended  that  an 
intensive  training  program  be  established  in  order  to  teach  Sis  how  to  use 
AIFs  more  effectively.  These  recommendations  apply  to  Phase  III  as  well, 
for  it  is  clear  that  many  Sis  have  not  yet  fully  explored  the  existing 
instructional  capabilities  of  their  ATDs. 

The  principles  of  effective  AIF  use  still  need  to  be  specified.  Such 
principles  will  not  be  derived  from  surveys  but,  rather,  from  empirical 
investigations.  Several  reports  by  R.  G.  Hughes  and  his  colleagues 
(Bailey  &  Hughes,  1980;  Bailey,  Hughes,  &  Jones,  1980;  Hughes,  1979; 

Hughes,  Hannon,  &  Jones,  1979;  Hughes,  Lintern,  Wightman,  4  Brooks,  1981) 
do  provide  conceptual  models  for  AIF-based  simulator  training  programs  and 
present  experimental  evidence  aimed  at  determining  the  training  value  of 
particular  features,  but  much  additional  work  is  needed  if  military  ATDs 
are  to  be  more  than  mere  substitute  weapon  systems. 

It  is  recommended  that  future  procurement  of  AIFs  be  preceded  by  a 
detailed  front  end  analysis  that  clearly  relates  AIF  capability  to  training 
needs.  The  analysis  should  consider  all  known  training  applications  of  the 
simulator  as  well  as  any  major  constraints  in  the  operational  environment. 
During  procurement,  AIF  specifications  should  be  prepared  so  as  to  meet 
user  needs  and  ensure  equipment  reliability.  After  operational  deployment, 
the  user  should  provide  adequate  instructor/operator  training  in  AIF  use. 
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PHASE  III  INSTRUCTIONAL  FEATURES  QUESTIONNAIRE 
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ADVANCED  INSTRUCT  ICNAL  FEATURES  -  SWI  SURVEY 


Name 


Rank _ Squadron _ Date 


FLYING  EXPERIENCE: 
Aircraft 


Total  Hours 


Instructor  Hours 


SIMJLATQR  EXPERIENCE: 
Simulator 


Total  Hours 


Instructor  Hours 


1.  What  percent  of  your  initial  instruction  on  simulation  training  consisted  of  formal 
classroom  instruction  and  what  percent  consisted  of  informal  instruction? 

_ %  formal  classroom  _ %  informal 

2.  Have  you  had  refresher  training  on  simulation  operation?  _  yes  _  no 

(If  no,  skip  next  two  items.) 

a.  How  long  has  it  been  since  you  last  had  refresher  training?  _ weeks 

b.  What  percent  of  your  refresher  training  was  formad  and  v^at  percent  informal? 

_ %  formal  classroom  _ %  informal 

BRIEFLY  DESCRIBE  A  "TYPICAL"  TRAINING  SESSION  CN  THIS  SIMULATOR: 


GQIERAL  CCMIENrS  AM>/QR  REQQM1EMDATIGN5 : 
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Read  the  definitions  of  each  instructional  feature  carefully,  in  the 
space  next  to  each  feature,  write  the  single  number  corresponding  to  the 
statement  that  best  describes  the  operational  status  of  that  feature: 

0.  The  simulator  has  no  such  capability. 

1.  Capability  present  but  I  have  never  seen  it  operate. 

2.  Capability  present  but  unreliable. 

3.  Capability  present  and  reliable. 


Instructor  Tutorial  -  provides  the  instructor  with  self-paced 
programned  instruction  in  the  capabilities  and  use  of  the  simulator. 

Reset  -  permits  instructor  to  "return"  the  simulated  aircraft  to  a 
stored  set  of  conditions  and  parameters. 

Total  System  Freeze  -  permits  instructor  to  interrupt  and  suspend 
simulated  flight  by  freezing  all  system  parameters. 

Partial  Freeze  -  permits  instructor  to  freeze  veurious  flight 
parameters  or  parameter  combinations  such  as  altitude,  heading, 
position,  attitude,  flight  system,  etc. 

Recorded  Briefing  -  permits  instructor  to  provide  student  with 
information  about  a  structured  training  session  through  audio/visual 
media  presentation. 

Denonstration  -  permits  instructor  to  demonstrate  optimal  electronic 
warfare  procedures  by  prerecording  and  subsequently  playing  back  a 
simulated  engagement. 

Record/Playback  -  permits  instructor  to  record  and  subsequently 
playbcKtk  a  segment  of  simulated  flight. 

Automated  Metlf unction  Insertion  -  permits  instructor  to  pre-program  a 
sequence  of  aircraft  component  malfunctions  and/or  emergency 
conditions. 

Hard  Copy  -  provides  a  record  of  alphanumeric  and/or  graphic 
performance  data  for  debriefing  purposes. 

Programmed  Threat  Control  -  conputer -controlled  standardized  training 
sessions  based  on  pre-progranmed  event  sequences. 

Manual  Threat  Control  -  permits  instructor  to  modify  threat  scenarios 
during  a  training  session. 

Procedures  Monitoring  -  permits  instructor  to  monitor  discrete  actions 
performed  by  the  student  in  accordance  with  a  procedurally  defined 
checklist. 

Parameters  Monitoring  -  permits  instructor  to  monitor  vaurious 
instrument  readings,  control  settings,  edrcraft  states,  or 
navigational  profiles. 

Electronic  Warfare  Performance  Scoring  -  provides  a  performance  metric 
that  suninarizes  the  outcomes  of  EW  engagements. 


) 


1.  During  Civt  typical  aiaaiona.  hew  often  did  you  uae  each  inatruetional  faatuctf  (Oiaek  the  wprepriata  vaea. 

/  /  /  Once  Bvary  S  /  OnB»  every  /  Onoa  a  /  2-4  TUm  /  S-7  Timm/  8  or  Mora  / 

/  tVk/  Navar  /  Miaaiena  or  baaa  /  2-4  Miaaiona  /  Niaaion  /  a  Mtaaion  /  a  Maaion/  Tiaea  a  Niaaion  / 


inatructor  Tutoriai  / 


V". 


id  on  tiw  dittnltlona  iloni  and  mt  your  axpiilineit  hew  ootintiillv  uMful  li  aheh  instrueUaml  fMture.  Rate  Mch 
.  JMMM  MCb  U  iVMUy  MV  to  UM.  (OiKk  tlM  iivcoprUM  tpaoi.) 


/  Hot  /  SUqhUy/  Fairly  /  Hodiratily/  Vity  /  ExtraMly  /  / 

/  UMful  /  UMful  /  UMful  /  UMful  /  UMful  /  UMful  /  IndiipinMbli  / 
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FB-lllA  OPERATIONAL  FLIGHT  TRAINER:  MEAN  RATINGS 
(AND  STANDARD  DEVIATIONS)  OF  AIF  UTILITY  AND 
UTILIZATION  BY  RADAR  NAVIGATOR  INSTRUCTORS 


Table  B2.  FB-lllA  Operational  Flight  Trainer: 

Mean  Ratings  (and  Standard  Deviations) 
of  the  Ease  of  AIF  Use 


Feature 

Transition 

Operational 

Ml  rill irm 

PF 

6.3 

6.2 

6.2 

(0.9) 

(0.9) 

TSF 

6.0 

6.1 

6.1 

(1.1) 

(0.8) 

R 

4.9 

5.7 

5.4 

(1.6) 

(1.0) 

PTC 

4.4 

4.5 

4.5 

(1.3) 

(1.4) 

PAM 

5.0 

4.1 

4.4 

(1.3) 

(1.3) 

PRM 

4.6 

3.6 

4.0 

(1.7) 

(1.6) 

RP 

3.9 

3.4 

3.6 

(1.4) 

(1.1) 

D 

3.1 

3.3 

3.2 

(1.6) 

(1.0) 

HC 

2.0 

3.0 

2.7 

(0.8) 

(1.8) 

Unweighted  Means 


4.5 


4.4 


4.5 


Table  B3.  FB-lllA  Operational  Flight  Trainer:  Mean 

Ratings  (and  Standard  Deviations)  of  the  Amount 
of  Training  Received  in  AIF  Use 


Feature 


Transition 


Feature 


Transition 


Operational 


Combine! 


